Rdf ingestion mvp #15741

stephengoldbaum · 2025-12-19T18:36:23Z

Summary

This PR introduces a new RDF ingestion source for DataHub, enabling ingestion of RDF/OWL ontologies (Turtle, RDF/XML, JSON-LD, N3, N-Triples) with a focus on business glossaries. The source extracts glossary terms, term hierarchies, and relationships from RDF files using standard vocabularies like SKOS, OWL, and RDFS.

What's New

Core Features

RDF Ingestion Source (type: rdf) - Native DataHub plugin for RDF/OWL ontologies
Multiple Format Support - Turtle, RDF/XML, JSON-LD, N3, N-Triples
Flexible Source Loading - Files, directories (with recursive option), URLs, and comma-separated file lists
Glossary Term Extraction - Converts skos:Concept and owl:Class to DataHub GlossaryTerms
Glossary Node Hierarchy - Auto-creates glossary nodes from IRI path hierarchies
Term Relationships - Extracts skos:broader and skos:narrower relationships as isRelatedTerms
Stateful Ingestion - Full support for stale entity removal via stateful_ingestion config
Platform Instance Support - Configurable platform instances via platform_instance config

Architecture

Modular Entity Processing - Clean separation with extractors, converters, and MCP builders
Dependency-Based Processing - Topological sort for correct entity processing order
Test Connection Support - Implements test_connection() for connection validation

Capabilities

The source supports the following DataHub capabilities:

Capability	Status	Notes
Glossary Terms	✅	Enabled by default
Glossary Nodes	✅	Auto-created from IRI path hierarchies
Term Relationships	✅	Supports `skos:broader` and `skos:narrower`
Detect Deleted Entities	✅	Requires `stateful_ingestion.enabled: true`
Platform Instance	✅	Supported via `platform_instance` config
Extract Descriptions	✅	Enabled by default (from `skos:definition` or `rdfs:comment`)
Data Domain	❌	Not applicable (domains used internally for hierarchy)
Dataset Profiling	❌	Not applicable
Extract Lineage	❌	Not in MVP
Extract Ownership	❌	Not supported
Extract Tags	❌	Not supported

Testing

Test Coverage

126 unit tests - Comprehensive coverage of core functionality, error handling, and edge cases
16 integration tests - End-to-end testing with golden file validation
Test scenarios include:
- Simple glossary ingestion
- Glossary with relationships
- Glossary with domains
- Multiple RDF formats (Turtle, RDF/XML, JSON-LD)
- Recursive directory ingestion
- Export filtering (export_only, skip_export)
- Stateful ingestion with stale entity removal
- Error handling (missing files, malformed RDF, invalid formats)
- Large file performance warnings
- Path traversal protection

Test Files

tests/unit/rdf/ - Unit tests for individual components
tests/integration/rdf/ - Integration tests with golden file validation
All tests passing ✅

Documentation

User Documentation

docs/sources/rdf/rdf.md - Comprehensive user guide (489 lines)
- Quickstart guide
- Configuration reference
- RDF format and source types
- Dialects and selective export
- Stateful ingestion guide
- Example RDF files
- IRI-to-URN mapping
- Glossary node hierarchy
- Supported vocabularies
- Limitations and troubleshooting

Recipe Examples

docs/sources/rdf/rdf_recipe.yml - Example recipes for basic and stateful ingestion

Integration Test Documentation

tests/integration/rdf/README.md - Detailed guide for running integration tests

Configuration Example

source:
type: rdf
config:
source: ./glossary.ttl
format: turtle
environment: PROD
stateful_ingestion:
enabled: true
remove_stale_metadata: true
export_only:
- glossary## Files Changed

Technical Notes

Security & Performance

URL Loading Security - Timeout limits, size limits, and redirect limits for safe URL loading
Path Traversal Protection - Configurable enforcement to prevent access outside intended directories
Memory Efficiency - Generator patterns for work unit generation and streaming for large URL downloads
Format Validation - Validates RDF formats before processing

Code Quality

Thread-Safe Registry - Entity registry uses double-checked locking pattern for thread safety
Component Validation - Validates registered components for entity type consistency
Type Safety - Complete type hints with proper forward references for MCP return types
Error Handling - Granular error reporting with structured logs and context
URN Generation - Standardized URN format using dot notation, proper encoding, and GUID fallback for non-ASCII characters

New Files

src/datahub/ingestion/source/rdf/ingestion/rdf_source.py - Main source implementation
src/datahub/ingestion/source/rdf/core/rdf_loader.py - RDF loading utilities with security
src/datahub/ingestion/source/rdf/core/urn_generator.py - URN generation with encoding
src/datahub/ingestion/source/rdf/entities/base.py - Base interfaces for entity processing
src/datahub/ingestion/source/rdf/entities/registry.py - Thread-safe entity registry
docs/sources/rdf/rdf.md - User documentation
docs/sources/rdf/rdf_recipe.yml - Recipe examples
tests/integration/rdf/test_rdf_source.py - Integration tests
tests/unit/rdf/ - Unit tests (multiple files)

Modified Files

setup.py - Added RDF source to entry points (line 862)

Breaking Changes

None - This is a new feature addition with no breaking changes to existing functionality.

Support Status

The RDF source is marked as INCUBATING (SupportStatus.INCUBATING), indicating it's ready for community adoption but may have minor version changes in future releases based on feedback.

Checklist

github-actions · 2025-12-19T18:36:36Z

Linear: ING-1308

codecov · 2025-12-19T18:38:58Z

Codecov Report

❌ Patch coverage is 81.14210% with 426 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...tahub/ingestion/source/rdf/ingestion/rdf_source.py	71.42%	122 Missing ⚠️
...rc/datahub/ingestion/source/rdf/core/rdf_loader.py	79.21%	95 Missing ⚠️
...ion/source/rdf/entities/glossary_term/extractor.py	59.11%	83 Missing ⚠️
.../src/datahub/ingestion/source/rdf/dialects/fibo.py	79.56%	19 Missing ⚠️
.../datahub/ingestion/source/rdf/entities/registry.py	90.60%	17 Missing ⚠️
...source/rdf/entities/glossary_term/urn_generator.py	68.18%	14 Missing ⚠️
...n/source/rdf/entities/glossary_term/mcp_builder.py	88.34%	12 Missing ⚠️
...datahub/ingestion/source/rdf/core/urn_generator.py	91.79%	11 Missing ⚠️
...c/datahub/ingestion/source/rdf/dialects/bcbs239.py	79.54%	9 Missing ⚠️
...rc/datahub/ingestion/source/rdf/dialects/router.py	87.09%	8 Missing ⚠️
... and 8 more

📢 Thoughts on this report? Let us know!

alwaysmeticulous · 2025-12-19T18:42:27Z

✅ Meticulous spotted 0 visual differences across 951 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit 8c9e53d. This comment will update as new commits are pushed.}

codecov · 2025-12-19T18:45:28Z

Bundle Report

Changes will increase total bundle size by 2.94kB (0.01%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	29.55MB	2.94kB (0.01%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	2.94kB	19.38MB	0.02%

Files in assets/index-*.js:

./src/app/ingestV2/source/builder/sources.json → Total Size: 39.44kB
./src/app/ingestV2/source/builder/RecipeForm/constants.ts → Total Size: 12.11kB
./src/app/ingestV2/source/builder/constants.ts → Total Size: 7.46kB
./src/app/ingestV2/source/builder/RecipeForm/rdf.ts → Total Size: 2.86kB

….Class and RDFS.Class - Updated the logic in `GenericDialect` to exclude ontology construct types while allowing OWL.Class and RDFS.Class to coexist with SKOS.Concept, enhancing compatibility with RDF standards.

- Updated the rdflib dependency in setup.py to specify a version range of >=6.0.0,<7.0.0, ensuring compatibility with existing RDF handling features.

- Updated the rdflib dependency in setup.py to specify an exact version of 6.3.2, ensuring compatibility with existing RDF handling features and preventing potential issues with future releases.

…er, and URN generator - Introduced new unit tests for various edge cases in RDF dialects (Generic, FIBO, Default), including handling of empty graphs, missing labels, and special characters. - Added tests for the RDF loader to cover format validation, file handling, URL loading, and zip file scenarios. - Implemented edge case tests for the URN generator, focusing on IRI parsing and platform normalization. - Enhanced overall test coverage to ensure robustness and reliability of RDF processing components.

… requests_file - Modified the RDF plugin dependencies in setup.py to add specific versions of requests (2.32.5) and requests_file (3.0.1) alongside rdflib (6.3.2), ensuring compatibility and stability for RDF processing.

…n in documentation generation - Added checks and warnings for missing platforms and plugins when processing README and documentation files, ensuring robustness in the documentation generation process. - Improved logging to provide clearer feedback when encountering issues with platform or plugin names during the generation of custom documentation.

…tion generation - Added a new metric for tracking missing capability data in the PluginMetrics class. - Changed error logging to a warning when a plugin is not found in capability data, incrementing the new metric instead of the failed count. - Updated exit behavior to only return an error code for actual failures, enhancing the robustness of the documentation generation process.

- Added new capabilities for ABS Data Lake, including support for containers, data profiling, and tags. - Enhanced Athena source capabilities with additional features such as lineage fine, schema metadata, and test connection. - Updated platform details and support statuses for both ABS and Athena sources. - Removed outdated entries and streamlined the capability structure for clarity.

…and URN generator - Introduced extensive unit tests for the RDF loader, covering URL loading, error handling, and path traversal protection. - Added tests for the EntityRegistry, focusing on registration methods, CLI name mapping, and processing order. - Implemented tests for the URN generator, addressing IRI parsing, platform derivation, and structure preservation. - Enhanced overall test coverage to ensure robustness and reliability of RDF processing components.

…ests - Updated the test cases in `test_registry_comprehensive.py` to include type hints for the `EntityProcessor` instantiation, improving code clarity and type safety. - Ensured consistency in the test setup for better maintainability and readability.

- Introduced extensive unit tests for the RDF loader, enhancing coverage for format validation, error handling, and file/URL detection. - Added comprehensive tests for the URN generator, focusing on platform normalization, IRI parsing, and path derivation edge cases. - Improved overall test coverage to ensure robustness and reliability of RDF processing components.

… tests - Updated unit tests in `test_urn_generator_additional_coverage.py` to include type ignores for invalid type checks in platform normalization, IRI path derivation, and group name generation methods. - Enhanced error handling assertions to maintain clarity while addressing type-related warnings from static analysis tools.

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX labels Dec 19, 2025

github-actions bot added the community-contribution PR or Issue raised by member(s) of DataHub Community label Dec 19, 2025

github-actions bot deployed to datahub-wheels (Preview) December 19, 2025 18:38 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) December 19, 2025 18:41 View deployment

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 19, 2025

vercel bot deployed to Preview December 19, 2025 18:48 View deployment

stephengoldbaum force-pushed the rdf-ingestion-mvp branch from 83de506 to 4ba2ba2 Compare January 6, 2026 17:38

github-actions bot deployed to datahub-wheels (Preview) January 6, 2026 17:40 View deployment

vercel bot had a problem deploying to Preview January 6, 2026 17:46 Failure

stephengoldbaum force-pushed the rdf-ingestion-mvp branch from 4ba2ba2 to 0238c98 Compare January 18, 2026 01:20

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 01:22 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 01:25 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 01:27 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 03:35 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 03:38 View deployment

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 03:41 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 03:45 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 03:46 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 04:00 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 04:04 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 04:05 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 04:09 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) January 18, 2026 04:13 View deployment

vercel bot had a problem deploying to Preview January 18, 2026 04:14 Failure

github-actions bot deployed to datahub-wheels (Preview) January 18, 2026 04:34 View deployment

stephengoldbaum added 7 commits January 31, 2026 21:05

fix(rdf): restrict rdflib version to ensure compatibility

ad5fa52

- Updated the rdflib dependency in setup.py to specify a version range of >=6.0.0,<7.0.0, ensuring compatibility with existing RDF handling features.

fix(rdf): pin rdflib version to 6.3.2 for compatibility

c531592

- Updated the rdflib dependency in setup.py to specify an exact version of 6.3.2, ensuring compatibility with existing RDF handling features and preventing potential issues with future releases.

stephengoldbaum force-pushed the rdf-ingestion-mvp branch from bfdc088 to 545aad0 Compare February 1, 2026 02:14

github-actions bot deployed to datahub-wheels (Preview) February 1, 2026 02:16 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) February 1, 2026 02:19 View deployment

vercel bot had a problem deploying to Preview February 1, 2026 02:29 Failure

github-actions bot deployed to datahub-wheels (Preview) February 1, 2026 02:32 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) February 1, 2026 02:36 View deployment

vercel bot deployed to Preview February 1, 2026 02:46 View deployment

github-actions bot deployed to datahub-wheels (Preview) February 1, 2026 04:27 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) February 1, 2026 04:31 View deployment

vercel bot deployed to Preview February 1, 2026 04:41 View deployment

github-actions bot deployed to datahub-wheels (Preview) February 1, 2026 13:36 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) February 1, 2026 13:40 View deployment

github-actions bot deployed to datahub-wheels (Preview) February 1, 2026 13:50 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) February 1, 2026 13:54 View deployment

vercel bot deployed to Preview February 1, 2026 14:04 View deployment

github-actions bot deployed to datahub-wheels (Preview) February 1, 2026 14:06 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) February 1, 2026 14:10 View deployment

vercel bot deployed to Preview February 1, 2026 14:20 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rdf ingestion mvp #15741

Rdf ingestion mvp #15741

stephengoldbaum commented Dec 19, 2025

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

alwaysmeticulous bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 19, 2025 •

edited

Loading

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rdf ingestion mvp #15741

Are you sure you want to change the base?

Rdf ingestion mvp #15741

Conversation

stephengoldbaum commented Dec 19, 2025

Summary

What's New

Core Features

Architecture

Capabilities

Testing

Test Coverage

Test Files

Documentation

User Documentation

Recipe Examples

Integration Test Documentation

Configuration Example

Technical Notes

Security & Performance

Code Quality

New Files

Modified Files

Breaking Changes

Support Status

Checklist

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alwaysmeticulous bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Dec 19, 2025 •

edited

Loading

alwaysmeticulous bot commented Dec 19, 2025 •

edited

Loading

codecov bot commented Dec 19, 2025 •

edited

Loading